Analysis of the Evaluation of Application-Led Research in Pervasive Computing

Pervasive computing researchers typically conduct their research through the development of prototype applications. Such research is motivated by a well-defined problem and evaluated by assessing the impact of deployed solutions. Accordingly, the evaluation phase assumes a critically important role in this process. Failure to sufficiently evaluate an application can have wide ranging negative effects. Differences between the pervasive computing and standard desktop paradigms preclude the use of established evaluation techniques without significant modification. In this paper we present a survey of the state of the art in pervasive application evaluation. We discuss the prominent challenges in conducting a pervasive computing evaluation and assess how the surveyed application evaluations have been affected by these challenges. We make recommendations for researchers conducting hypothesis-led pervasive technology evaluations.